DOCUMENT RESUME 



ED 365 940 



CS Oil 532 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Lemke, J* L* 

Multiplying Meaning: Literacy in a Multimedia World. 

Draft. 

Dec 93 

16p.; Paper presented at the Annual Meeting of the 
Nat i onal Reading Conference (43rd, Charleston, SC, 
December 1-4, 1993) . 

Viewpoints (Opinion/Position Papers, Essays, etc.) 
(120) " Speeches/Conference Papers (150) 

MFOl/PCOl Plus Postage. 

^Communication (Thought Transfer) ; Futures (of 
Society) ; Higher Education; Hypermedia; '^Literacy; 
''^Reading Processes ; Semiotics ; '^Text Structure 
Discourse Communities; '^Meaningf ulness ; Multimedia 
Technology; ''^Text Factors ; Text Processing 
(Reading) 



ABSTRACT 

As material objects, texts are as much the product of 
visual semiotic codes as of linguistic ones. And throughout history, 
verbal texts have been combined with nonverbal, visual modes of 
presenting information, taking a stance toward information and 
readers, and organizing parts into wholes. The major challenge to 
creating multimodal texts in the near future will be a lack of 
multimedial literacy. A more fundamental understanding of existing 
cultural conventions in communities for combining verbal and 
nonverbal elements in multimedial texts is needed. To understand how 
meaning is made simultaneously in several semiotic modalities, common 
features of all semiotic systems must be identified, i.e., the 
presentational, the orientational , and the organizational features. 
Scientific and technical texts have long preserved a tradition of 
incorporating nonverbal visual-graphic elements as integral and 
normal parts of their genres. What it means to "read" a text of this 
kind depends on the literacy practice involved; that is, on the 
cultural activity as part of which meaning is being made with this 
text. (Contains 31 references.) (NH) 
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Literacy practices are the ways we make meaning with texts. If they 
are to be useful, practical skills, then they must be adapted to the 
real diversity of actual texts in the many communities to which we all 
belong* They must function within the many contexts of the actual ac- 
tivities in which texts play a part in our communities. For this rea- 
son alone, literacy practices cannot be usefully understood as merely 
linguistic practices, nor texts discussed as if they were made and in- 
terpreted only through the semiotic system of language. 

Written texts, as we usually encounter them, are visual (and tactile) 
objects. We construe the visual patterns we see in them as written 
words, but we know that those words are also presented to us in some 
particular typeface and font, and are laid out on the page in some 
visual arrangement that is often relevant to interpreting their mean- 
ings. Visual information, above and beyond what is necessary to con- 
strue linguistic units, but relevant to the total potential meaning of 
the text in its social context is always present. As material objects, 
texts are as much the product of visual semiotic codes as of linguis- 
tic ones. And throughout their history, they have combined verbal text 
with nonverbal, visual modes of presenting information, taking a 
stance toward information and readers, and organizing parts into 
wholes . 

Across all genres of texts, in all registers, in all the uses of texts 
in human social activity, in all our communities and subcommunities , 
lists, cables, diagrams, pictures, drawings, graphs, and photographs 
are common, normal, and essential elements of the text as a whole. The 
total information available in the text which is relevant to and 
normally used in literacy activities involving that text depends as 
much or more on non-verbal visual elements and codes as on the verbal 
elements themselves . 
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multimodal communication is historically, developmentally , and practi- 
cally what human beings are most at home with. 

The simple technologies for single-copy texts often make it as easy to 
draw as to write. Printing initially required separate processes for 
moveable type and pictorial material, much as some early computer sys- 
tems separated text from graphics. Later forms of printing, and then 
xerography, made it possible to multiply copies of multimodal 
originals. When those originals were made by the typewriter, separate 
and rather difficult processes were needed tc add pictorial material 
of comparably prestigious forms. Now computer technology is making it 
possible for all of us to compojse multimedia texts, and to copy or 
distribute these with ease. True multimedia texts can now include not 
just verbal text, visually organized and highlighted (by size and 
font, placement, color, etc.), and depictive graphics {drawings, 
diagrams, graphs, photographic images, 3D renderings), but audio 
(music, voice, recordings, synthesized sound), video (created, re- 
corded, filmed, stills and full-motion) , and animation (of both text 
and images) . 

The major challenge to creating multimodal texts in the near future 
will be our own lack of multimedia literacy. We need to have a more 
fundamental understanding of multimodal communication, and particular- 
ly of the existing cultural conventions in our communities for combin- 
ing verbal and nonverbal elements in multimedia texts. We need to be- 
gin to extend those conventions to take full advantage of the communi- 
cative resources which our new technologies are giving us. We need a 
general semiotic understanding of multimodal communication and multi- 
media literacy. Let me try to sketch out some of the elements of such 
a picture. 

Multimodal Communication 

As the ways we make meaning with texts, literacy practices are social 
and cultural practices in the sense that they are primarily character- 
istic of the communities we belong to and are only slightly distinc- 
tive from one individual to another. As social practices they are al- 
ways embedded in social situations, in particular kinds of culturally 
recognizable activities. These activities are physical, material pro- 
cesses, involving bodies and other objects of the ecosystems to which 
our community belongs. They are part of the system of interdependent 
processes which constitute the ecosystem and the community. But as 
cultural activities, these processes also have meanings for us, and we 
enact them according to what they mean to us . As a result, the rela- 
tions among our physical and material actions reflect and reinforce 
the relations of meaning that we attribute to them. For these reasons 
it is not possible to understand the material ecosystem and the cul- 
tural semiotic separately: they form a unified ecosocial system (Lemke 
1993a) . 

We need to understand literacy practices as part of an ecosocial sys- 
tem. They are semiotic practices, meaning-making practices, character- 
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istic of a community (Lemke 1989a) ; and they are also material pro- 
cesses of interaction that embed our bodies in larger ecosocial sys- 
tems, and without which we could not exist (cf. Lemke 1984, forthcom- 
ing) . As integral parts of ecosocial systems, people and their liter- 
acy activities, are dynamic: they extend over time along the trajec- 
tories we call developmental pathways. Here, most clearly, our 
biological development and our behavioral development are visibly in- 
terdependent. Literacy practices develop with age and experience, and 
they evolve with time and cultural change. Their evolution depends on 
the diversity of literacy practices from one community to another. 

Because literacy practices are processes of an ecosocial system, they 
cannot be understood solely in terms of what happens within any one 
individual, or in terms of what is happening at any one moment in de- 
velopmental or historical time, and they cannot be understood as pure- 
ly linguistic, cognitive, or semiotic processes apart from their 
material embodiment . 



The materiality of literacy practices insures that they cannot be ade- 
qucitely understood in terms of any one semiotic modality, such as lan- 
guage. Languages, or linguistic semiotic resource systems, are 
analytical abstractions from embodied social practices: from speakings 
and writings and the activities that provide the contexts on which 
their cultural meanings depend. Ycu cannot utter a lexical item, a 
morpheme, or a phoneme; you can only utter an acoustical sound stream 
replete with linguistically non-distinctive features, whose segmenta- 
tion and classification into meaningful units is a miracle of coupled 
processes from the neurological to the cultural within an evolved, de- 
veloped, self -organized ecosocial system of phenomenal complexity. The 
same word (as an abstracted linguistic unit) still sounds recognizably 
different in my mouth and in yours; and it cannot exist at all unless 
it is more than a linguistic word, unless it is also the unique sound 
made by some material system. Acoustic patterns that we can construe 
in that soundstream not only tell uis what word of our language it most 
likely is, but a lot about the person who uttered it that may not be 
relevant to what word it is, but is very relevant to our communication 
with the person. 



In face-to-face communication, we not only utter soundstreams , but we 
dance with one another: we move our bodies, from our eye-gaze and eye- 
blinks to our arm and hand movements, our body postures, our leanings 
toward and away from one another, in a complex interactional synchrony 
of which the soundstreams we make are an integral part (cf . Kendon 
1990, Scheflen 1975). Meaningful units or meaningful acts, whether 
words or culturally recognizable gestures , are abstractions from this 
action-stream. Language co-evolved with other modes of meaning-making; 
it evolved to function in co-ordination with our construction of mean- 
ing, in semiotically-mediated action and active perception. Language 
never occurs as a pure form isolated from other kinds of meaning. In 
both face-to-face oral communication and in writing, the semiotics of 
visual and kinesthetic information is co-dependent with that of lan- 
guage. Our perception and interpretation of oral l^^nguage and of writ- 
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ing also requires that we combine visual and motor processes with lin- 
guistic skills • Communication is originally, in evolution and in de- 
velopment, a uj3i tary process • It is our cultural analytical practices 
that teach us to separate speech from other aspects of unified bodily 
processes, to separate speech from gesture, and gesture from writing, 
and writing from depiction. 

Writing and drawing share common ancestry; each generation must be 
taught anew to make the separation between them. Writing is not merely 
the annotation of speech, as drawing is not simply the inking of im- 
ages . Drawing begins as the extension to paper of gesture , as the pro- 
duct of lasting visual traces of our gestures in acts which are indis- 
criminately gesturings and drawings (cf. Arnheim 1956, cited in Kress 
& van Leeuwen 1990: 25). And these are accompanied you can be sure by 
vocalizations which are not as distinct from other gestures as the ab- 
stractions of linguistics and the ideology of intellectual verbalism 
dispose us to believe. It is not surprising that children do as one 
act what adults have been taught to separate into two: drawing and 
writing (cf. Dyson 1991; Hicks & Kanevsky 1992). Like our first draw- 
ing, our first writing is not a representation of speech, but an 
extension of it that produces a lasting visual trace. 

In the progressive cultural differentiation of the action-stream of 
communication and the its lasting traces into speech, gesture, writ- 
ing, and drawing there is a loss of unity, but there is also a semi- 
otic gain. By a general process, common in development and evolution, 
sometimes called semogenesis (cf. Halliday 1992), when an original 
unity becomes semiotically construable as being composed of separable 
units, those units are now free to combine in new ways, making new 
meanings possible, and the culture is free to evolve new restrictions 
on these combinations, endowing them with yet another dimension of 
meaning (as "indices" of some subcommunity ; Lemke 1993b) . When writing 
is distinguished from drawing, it becomes possible to compose multi- 
media texts, incorporating both semiotic modalities, and allowing us 
to multiply the meanings made with one by those made with the other, 
provided our community has established conventions for how to read 
such texts across modes. 

^v^ery written text combines both linguistic and other, visual semiotic 
modalities (e.g. typographical ones). It must, if only because our 
material interaction with the text always makes it more than just a 
linguistic, or even visual, abstraction. It is always more for us than 
just an abstract graphological design, or an abstract verbal text. It 
is real paper (or stone, or computer screen), with real ink diffusing 
through the strands of cellulose, construable as writing with some 
particular typeface, in a particular font and point-size, laid out on 
the page in some particular visual arrangement — all potentially 
highly relevant to what it means, or can mean, for us, not just lin- 
guistically, but typographically and even f orensically . It is the 
materiality of the text as object, or more precisely of our ways of 
interacting with it as part of a literacy activity, which insures that 
it cannot be comprehended within the limits of any one abstracted 
semiotic system, linguistic or visual. 
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Presentational , Orientational , and Orcfanizational Meaning 

In order to understand the underlying unity of human communication, to 
understand how we make meaning simultaneously in several semiotic 
modalities (language, gesture, picture, etc), we need to identify the 
common features of all semiotic systems. Once we can see all forms of 
symbolic expression in the same terms, we can then begin to understand 
how our capacity to make our meanings specific arises from our prac- 
tices of distinguishing and combining, of mul tiplyiiig meanings made 
with one system of semiotic resources (e.g., language) by those' made 
with another (e.g., depiction). 

A long historical tradition in functional linguistics, extended ard 
generalized in the last few decades by Halliday (1978), points to 
three general sorts of meaning that we make in every linguistically 
meaningful utterance. Halliday 's work has identified the specific lin- 
guistic resources by which we make these meanings, and has emphasized 
what was implicit in earlier accounts: that we make all of them simul- 
taneously in every utterance. My own work (Lemke 1989b, 1990, 1992) 
has extended Halliday 's three linguistic metaf unctions in the context 
of a general social semiotics of action (Lemke 1984, 1993a, forthcom- 
ing) to three generalized semiotic functions v.aich I believe we neces- 
sarily also enact in every meaningful action we perform. These three 
functions, or aspects of our meaning-making. Presentational r Orienta- 
tionalr and Organizational, define both the production and interpreta- 
tion of multimedia texts. They provide the unifying framework ve need 
to understand multimodal communication and the multiplication c-f mean- 
ing in multimedia texts. 

Every meaning-making act constructs a Presentatioxnal "state-of- 
affairs" that construes relations among semiotic participants and pro- 
cesses as if they were being observed, objectively and synopt j cally , 
from some outside vantage point. In language, this is the so-called 
representational or prepositional function, the sense in whicn we 
speak "about" something, construct a theme or topic, make predications 
and arguments. In less philosophical and more linguistic ter^ns, this 
means that we deploy the resources of grammar and lexis to specify 
some process or relationship and its semantic participants (agents, 
patients, instruments, etc.) and circumstances (when, wherf;, why, how, 
under what conditions, etc.) In visual depiction, this is the figural 
or representational function that presents to us a scene i^hose ele- 
ments we can recognize and which have comprehensible relations to one 
another in terms of the typical scripts of that scene. This is what 
tells us what we are being shown, what is supposed to be "there", to 
be happening, or what relations are being constructed among the ele- 
ments presented. 

At the same time every meaning-making act constructs an Orientational 
"stance" toward that state-of -affairs , often to indicate how true or 
certain the producer wishes the interpreter to take it as being, or to 
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indicate an evaluation of it as good or bad, ordinary or suprising, in 
the perspective the producer is creating for the interpreter • But 
Orientational meaning goes naturally beyond this to also include the 
construction of a social relationship between producer and inter- 
preters (present or imagined) , and more generally a relative position- 
ing of the producer and text in the whole social space of possible 
discourses and viewpoints on the state-of -affairs . In language, this 
is the pragmatic or interactional function, through which we take a 
rol^ in the communication event and construct the nature of the 
"speech act" we are performing (informing, querying, commanding) and 
the social relationship to the addressee we are enacting (bullying, 
beseeching, promising, threatening) . It is always done in the context 
of larger social relations and groupings that transcend any particular 
communicative event or text, and against the background of the various 
other texts in the community constructed from other possible points of 
view (cf. Lemke 1985, 1993c; Bakhtin 1935/1981 on heteroglossia , and 
commentary in Lemke 1988a, Forthcoming) . 

In depiction, every image takes an Orientational stance which posi- 
tions the viewer in relation to the scene (e.g. intimate, distant; su- 
perior, subordinate) , establishes some sort of evaluative orientations 
of the producer/interpreter toward the scene itself (tragic, comic; 
normal, surprising), and does so against the background of other pos- 
sible viewpoints, and depictions, of similar scenes. 

Finally, every meaning-making act constructs a system of Organiza- 
tional relations defining wholes and parts of those wholes, both in 
the semiotic space of the text and in the (ecosocial) interactional 
space of the meaning-making act itself. Language creates words-in- 
phrases, phrases-in-clauses , chains of reference and cohesion, and 
larger and subtler structures and textures of the verbal text (Hal- 
liday's textual metaf unction , cf . Halliday & Hasan 1976, 1989; Lemke 
1988b, in press) , Conversation as an activity creates in-groups and 
out-groups, shifting dyads within larger groups, etc. Depiction 
deploys compositional resources to organize the visual text into ele- 
ments and regions, and to link disjoint regions by such features as 
color and texture. As material objects, depictions participate in in- 
teractions that define parts and unite them into wholes in the 
ecosystem where objects are viewed and used. 

In recent work, Michael O'Toole (1990) and, independently, Gunther 
Kress and Theo van Leeuwen (1990) have demonstrated these homologous 
dimensions of meaning in visual semiotic productions of many kinds: 
paintings, sculpture and architecture; photographs, advertisements, 
and cinema. Kress and van Leeuwen have also considered some of the 
parallels between text and drawings in children's writing (1990). 
Their work clearly shows the usefulness of taking a " trif unctional" 
perspective such as that sKetched here for visual semiotics, and the 
interesting similarities and enlightening differences between the 
specific meaning systems of these analytically distinct, and func- 
tionally and culturally specialized, semiotic resource systems and the 
genres we produce by deploying them, (O'Toole refers to represents- 
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tionalr modal r and textual dimensions; Kress and van Leeuwen to idea- 
tionale interpersonal, and compositional.) 

In an early effort (Lemke 1987) I analyzed how teachers and students 
made sense with each other by codeploying verbal, gestural r and pic- 
torial resources. I found that if one regarded each of these as con- 
stituting a separate "channel" of communication, that sometimes the 
same or equivalent information passed nearly simultaneously in more 
than one channel r sometimes the information in the two channels was 
complementary, and sometimes information came first in one channel, 
and later in another- It became very clear to me that the meanings 
that were being constructed were JOINT meanings produced in the inter- 
section of different semiotic systems. While it was useful to analyti- 
cally separate these into different "channels," there was also an un- 
derlying unity to the meanings produced. The separation neglected a 
fundamental unity of communicative meaning-making that made the co- 
ordination among channels not only possible, but normal. 

Since that time I have wanted to complement such work on joint visual- 
verbal meaning-making in spoken discourse (cf. Ochs et al. in press-a. 
in press-b) with studies of written text genres in which there are 
long and highly developed historical traditions of close co-ordination 
of verbal with visual elements. Since my own earlier work had studied 
the classroom communication of science (see Lemke 1990 and references 
therein) , where I had found that my experience as a "native" of this 
subculture had been extremely useful, I decided to begin with a study 
of the written and graphic genres of scientific and technical dis- 
course . 

This discourse community, because of its economic resources, famil- 
iarity with new technologies, and communicative needs, is already 
rapidly moving toward computer-based multimedia communication (cf . 
Tomer 1994 on the NCSA Mosaic and Collage systems) . I hope to follow 
its development there, and to extend the scope of my work to include 
educational anc scholarly computer-based multimedia and hypermedia 
texts more generally (Lemke 1993d) . 



Multiplying Meaning in Scientific Texts 

Scientific and technical texts have long preserved a tradition, which 
was somewhat suppressed in literary texts (cf. above), of incorporat- 
ing nonverbal visual-graphic elements as integral and normal parts of 
their genres. These texts normally contain such elements as drawings 
and visual renderings (often the representation of observations of the 
natural world) , schematic diagrams (abstracting theoretically critical 
features of a machine or complex system) , maps and time-series lists 
and charts (which, together with tables, were probably the forerunners 
of true quantitative graphs; cf. Tufte 1983); two-dimensional tables 
whose elements may be verbal, numerical, or graphic; and bar- and ^ 
line-graphs (representing quantitative codependence of variables in 
experiments or according to theoretical predictions) . Contemporary 
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texts frequently include photographs of items studied, and reproduc- 
tions, often in color, of computer models and simulations, frequently 
in three-dimensions. New technologies, such as CD-ROM and laser disks, 
as well as the computer-based multimedia and hypermedia systems al- 
ready mentioned, are adding to this mix audio and full-motion video 
elements as well as animations and simulations, again often in 3D. 

In a recent preliminary survey of 43 journal articles and book chap- 
ters occupying over 400 pages of text in several scientific and tech- 
nical research publications, there were, on average, at least one, and 
sometimes as many as 4, nonverbal visual displays per page of running 
text. This does not include mathematical expressions of which there 
were an average of an additional one to three (and up to as many as 7) 
per page, depending on subfield. 

It is an interesting question how we should count mathematical equa- 
tions that appear as part of a text, expecially when they are visually 
set off in page composition (usually standing along on their own line 
with whitespace above and below) . They are not strictly speaking part 
of the verbal text, though they can be read aloud in a variety of ways 
(by symbol names, in mathematical English, or in a mix of mathematical 
register and that of the technical register whose lexical items and 
semantic relations their symbols conventionally represent) . They are 
visual and nonverbal, and depend on a semiotic system which may be an 
extension of natural language semantically , but which is also one that 
has some degree of autonomy as a system of signs. 

In 20 articles (62 pages) comprising two issues of Physical Review 
Letters, a journal for brief communications of significant new experi- 
mental or theoretical results in all areas of physics, there were an 
average of 1.2 distinct graphic elements (graphs, diagrams, tables, 
etc.) per page of running text, and an additional 2.7 mathematical 
equations per page. It was not unusual to find 3 graphic elements on a 
single page, or 6 equations. Experimental articles have more graphic 
items, theoretical articles more equations, in general. 

In 31 articles (75 pages) comprising two issues of Science, a journal 
for brief communications of significant new findings mainly in the 
biological and earth sciences, excluding mathematics expressions and 
tables, there were 2.0 graphic elements per page on average, and rela- 
tively few equations (tables were common, about one per article) . 

In a later survey, looking at Science and its British twin Nature, for 
articles particularly rich in the diversity of their graphic elements, 
a sampling of 10 articles (total 40 pages), had an average of 3.6 such 
elements per page (excluding matl:ematical expressions and tables). 

A typical page of the texts of these genres shows a visual layout 
which is quite rich in its organizational complexity. It may contain 
headers and footers, with verbal text and numberings . It will usually 
have two or three vertical columns of the Main Text, incorporating oc- 
casional mathematical expressions set on their own lines, but often 
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punctuated as if grammatically part of the embedding sentences. These 
columns will be interrupted by tables, and suspended over portions of 
the page to accomodate larger figures containing one or several graph- 
ic elements, typically line graphs, diagrams, and photographs. Each of 
these will have; a Caption, sometimes as extensive as the main text on 
that page, which takes the form of verbal text, sometimes itself in- 
corporating mathematical equations . 

It is certainly not true that the verbal text contains all of the sig- 
nificant information presented in the article. Generally experimental 
data and many key relationships and patterns are presented only in 
tables or graphs, or in equations, and are not repeated in the verbal 
text. Much critical information is contained in the captions that is 
not mentioned in the Main Text. The information in the figures is 
generally not correctly or adequately interpretable without reference 
to the caption for that figure. There seems to be a convention that 
Figures-and-Captions , as a graphic-verbal unit, should be relatively 
self-contained. It is often not necessary to refer to the Main Text to 
understand what is presented in these units, though it is necessary to 
do so to find what the authors consider to be the significance of what 
they present. The Figure-and-Caption unit seems to express the ideol- 
ogy of this community that factual information can and should be sepa- 
rated from its interpretation. Figure-and-Caption units ten< to con- 
struct themselves as purely factual presentations. For this purpose. 
Tables are also Figures. 

What does it mean to read a text of this kind? We have to recognize 
that the answer depends on the literacy practice involved; that is, on 
the cultural activity as part of which meaning is being made with this 
text. In this community, for example, there is a literacy practice of 
"consulting*' an article in order to find a specific item of informa- 
tion, often a numerical value for some observation or a specif ic equa- 
tion (representing a theoretical model) . One may consult the article 
also to identify a particular "claim" by the authors (for purposes of 
citation or refutation) , but one may also consult it to look at the 
shape of a curve (as in a line graph) or the topological connectivity 
of a gene map, a model of a protein, or the basins of attraction of 
some complex dynamical system. In short, not only does one often look 
for purely visual-graphical information in the text, but in searching 
for any information one is likely to need to refer to both verbal text 
and graphical elements and construe the conventional relations between 
them appropriate to this genre. 

At the other extreme, there might be a complete reading" of the text, 
for the purposes of preparing to be tested on it or for debating its 
contents privately or publicly with its author. Such comprehensive 
readings of a text are very rare, I believe. The texts of these 
genres, unlike, perhaps, those of textbooks or of more synthetic ex- 
positions of a subject, are not really meant to be read linearly from 
start to finish. They are more like encyclopedia entries. One can read 
them completely, but one is more likely to read only parts of them. 
The most visually prominent elements of these texts are the Figures, 
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followed often by the Abstracts, and I believe these are the parts 
most often read* The least visually prominent are the References or 
citations to other articles, any one of which a reader might consult, 
but which no reader is likely to read comprehensively. 

Strangely, perhaps, Captions are visually prominent more by their 
length in some texts than by their other visual features. They are 
usually printed in typefaces much smaller than those of the Main Text, 
despite the fact that the information they contain is usually essen- 
tial to interpreting the Figures, and so to extracting what is for 
many readers the most important information in the text. Captions are 
supposed to be subordinated to their Figures, and Figures, by some ac- 
counts, are subordinate to Main Text, but in the dominant literacy 
practices of the community. Figures are very nearly co-equal with Main 
text in importance, and their Captions are an integral part of them. I 
think we can safely predict that in the era of hypertext links, the 
successors to Figure captions will normally be presented in the same 
size text as the verbal Main text, and that this will be quietly 
received in the community as an improvement in the genre. 

In a relatively complete or comprehensive reading of texts of this 
genre, a typical reader would begin with her point of entry to the 
taxt, usually a keyword in the title or the name of the author, which 
makes the text relevant to some task. The Abstract would be read cur- 
sorily as a preview of the text (it is read carefully only in order to 
decide whether to read it> . The visually prominent figures might at- 
tract attention and lead to a survey of them, with occasional 
references to the captions for clarification. If the specialty field 
is less familiar, the captions become essential to interpreting the 
figures. If the field is highly familiar (and remember that most 
readers of these texts are other experts in the specialty) , they will 
be read only for technical details, and probably later. One might scan 
the references at the end to see if one's own work is cited, or to see 
if reference is made to some other particular article, author, or re- 
search group. 

In reading comprehensively, the Main Text is only a spine. The main 
text itself frequently refers the reader to other material: most fre- 
quently to citations of references, or to the Figures-and-Captions . A 
caption may back-refer ("see main text"), or cross-refer ("see Figure 
X''), and may also cite references. This is already a miniature of the 
hypertext paradigm, in which various units of the total available text 
cross-refer at various points to one another, and in which there are 
many possible pathways through the textworld, not all, or even most of 
which, necessarily pass through every unit of text. Reading this genre 
is a highly non-linear process. 

The basic relations of Main Text to Figure, and of Figure to Caption, 
highlight the constant co-ordination of verbal meanings and visual- 
graphical meanings that are necessary to interpret the text. Even 
within a Figure, there are verbal and quasi-verbal (symbolic) "labels" 
that indicate the meanings of the abstract visual elements presented, 
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and which make it possible for verbal text to cross-refer to visual 
elements within a B^igure (*'as appears at point A of Figure 3"). What 
the words of the text mean depend on our being able to co-ordinate 
them with elements of the Figures, and the identity and interpretation 
of visual elements in the Figures depends on the accompanying verbal 
text ♦ 

Verbal meanings are more specific when they are coupled with visual 
elements in a Figure ("the relation between lines A and B in Figure 
4"), often in ways that cannot be equivalently specified in purely 
verbal terms (there is no vocabulary for describing the global rela- 
tionships between two arbitrary curved lines, for example) , but which 
are immediately apprehended by the visually trained specialist* And 
visual relations take on added, indeed more specific, significance, 
when they are contextualized by the verbal text. This is an instance 
of the general principle of multiplicative meaning in multimodal 
texts. If we have a set of words, and a set of pictures, that are not 
in one-to-one correspondence with each other, but which may be freely 
combined in different ways, then the set of possible meanings is en- 
larged to be the product (not the sum) of the meanings that can be 
made with each set separately. Any actual combination of a word and a 
picture is one instance drawn from this product, and so its meaning is 
specific in that higher degree. (Thus, if we had five words and four 
pictures, each word has a specificity of one in five, each picture a 
specificity of one in four, but each word-picture combination has a 
specificity of one in twenty.) 

In an idealized single semiotic modality, language as viewed by lin- 
guists, say, the three generalized semiotic functions or aspects of 
meaning in every utterance also combine in this way. So the meaning of 
an utterance is threefold specific: for its Presentational or 
thematic-propositional meaning, for its Orientational or interactive- 
attitudinal meaning, and for its Organizational or structural- 
compositional meaning. What happens when we have now two semiotic 
resource systems jointly deployed as in a multimedia text? 

We must first recognize that it is not true in practice that all pos- 
sible combinations of meaning elements are equally likely. If this 
were so there would be no culture. Cultural patterns are the skewed 
probability distributions (metaredundancy relations, cf. Lemke 1984) 
that tell us that if we find one meaning element in some context then 
we are more likely to find some others with it, and that if we find 
some combinations together, we are more likely to be in a particular 
context (and a particular subculture). Genres are such patterns, 
viewed synoptically , as the product of changing likelihoods of transi- 
tional probabilities (Halliday 1992, Lemke 1991): first this meaning 
element (or combination), then more likely this one than that one* 

Subject to these limitations, which invite us to specify the conven- 
tions of a particular genre in a particular field of discouri^e, we see 
that, to a first approximation, the Presentational meaning elements 
from each of, say, two semiotic modalities multiply each other to in- 
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crease the specifity of any possible Presentational meaning in the 
joint multimodal text, and the Orientational elements multiply one an- 
other si'nilarly, and so do the Organizational elements. 

Perhaps we should call this the second, rather than the first, approx- 
imation since there is a prior (physicists might call it the zeroeth 
approximation) relation: the multiplication of the Presentational, 
Orientational, and Organizational meanings of each unit within a 
single semiotic, in the units of the verbal text itself, or in the 
units of, say, the Figure itself. 

Is it then possible for, say, a Presentc.tional meaning element in a 
Figure to contribute to the Orientational meaning of the whole, cross- 
multiplying with an orientational meaning element in the verbal text? 
Indeed it is, at least insofar as there tend to be consistent patterns 
across metaf unctions . An alternative analysis, which has the virtue of 
making the relations appear simpler, would say that in this case the 
visual Presentational element is actually contributing as well to the 
Orientational meaning in its own semiotic because it is a convention 
of the genre that such Presentational elements tend to occur only with 
this particular Orientational meaning, and so their very occurence im- 
plies, within the genre, this Orientational meaning, which then com- 
bines by the original rule with the Orientational meaning elements of 
the other semiotic. 

Commonly, for instance, a graph of experimental data points will indi- 
cate with error bars the uncertainty in a measurement. This is 
Presentational information within the graphic semiotic insofar as it 
tells us what the estimated probable error of the measurement is. But 
it also contributes to the Orientational meaning of the graph as a 
whole, since it indicates the degree of certainty or uncertainty which 
the authors are constructing for their claim as to the numerical value 
of the measurement itself. By convention, large error bars indicate 
less certainty. We would then expect to find, in the verbal text, some 
characteristic signs of modalization in the corresponding claim, 
orientational meaning expressed by such markers as "it may be that" or 
"it is likely that" or even "we believe that" (in which "believe" is 
both a Presentational process, and an Orientational modality, con- 
trasting with both "understand" — a different process — and with 
"suspect", the same process, but with a lesser claim of certainty). 
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